Dockerを使わずにEC2上でQdrantを起動してみた

#Qdrant

#ベクトルデータベース

岩田智哉

2024.10.16

リテールアプリ共創部@大阪の岩田です。

最近ベクトルデータベースのQdrantを触って遊んでいます。公式ドキュメントのQuickStartなどではDockerを使ってパパっとQdrantを起動するやり方が紹介されているのですが、せっかくなのであえてEC2上で直接Qdrantを起動する方法を試してみました。

環境

今回利用した環境は以下の通りです。

EC2
- OS: Amazon Linux 2023.6.20241010
- AMI: al2023-ami-2023.6.20241010.0-kernel-6.1-x86_64
- インスタンスタイプ: t3.large ※最初t3.microやt3.mediumで試しましたが、Qdrantビルド時にメモリ不足で固まってしまいました。t3.largeでもOOM Killerが発動することがあったので、万全を期すならさらにインスタンスタイプを増強した方が良いかもしれません。
Qdrant
- バージョン: 1.12.1

Qdrantをビルドしてみる

EC2インスタンスを起動後にまずはRustのインストールを行います。

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

次にQdrantのソースをクローンするためにgitをインストールします。

sudo dnf install -y git

続いてQdrantのビルドに必要な依存関係をインストールします。公式ドキュメントには以下のように記載されており、必要な依存関係はDockerfileから読み取ってね。とのことです。

Before compiling, make sure that the necessary libraries and the rust toolchain are installed. The current list of required libraries can be found in the Dockerfile.

sudo dnf install -y clang git gcc g++ protobuf-compiler protobuf-devel

これでビルドの準備が整ったので、GitHubからQdrantのソースを取得します。

git clone --depth=1 https://github.com/qdrant/qdrant.git

クローンしたディレクトリ配下に移動して...

cd qdrant

いざビルドです。ここは結構な時間がかかります。

cargo build --release --bin qdrant

最終的に以下のように出力されました。

   ...略
   Compiling actix-web-extras v0.1.0
   Compiling actix-cors v0.7.0
   Compiling tower v0.5.1
   Compiling colored v2.1.0
   Compiling constant_time_eq v0.3.1
    Building [=======================> ] 782/790: librocksdb-sys(build)
   Compiling rocksdb v0.22.0
   Compiling collection v0.4.2 (/home/ec2-user/qdrant/lib/collection)
   Compiling storage v0.2.0 (/home/ec2-user/qdrant/lib/storage)
   Compiling qdrant v1.12.1 (/home/ec2-user/qdrant)
    Finished `release` profile [optimized] target(s) in 48m 54s

48分54秒でビルド完了です。

Qdrantを起動してみる

ビルドできたのでQdrantを起動してみましょう。

./target/release/qdrant

以下のように出力されればOKです。

           _                 _
  __ _  __| |_ __ __ _ _ __ | |_
 / _` |/ _` | '__/ _` | '_ \| __|
| (_| | (_| | | | (_| | | | | |_
 \__, |\__,_|_|  \__,_|_| |_|\__|
    |_|

Version: 1.12.1, build: 9c20e9e2
Access web UI at http://localhost:6333/dashboard

2024-10-16T02:34:56.828949Z  INFO storage::content_manager::consensus::persistent: Loading raft state from ./storage/raft_state.json
2024-10-16T02:34:56.829817Z DEBUG storage::content_manager::consensus::persistent: State: Persistent { state: RaftState { hard_state: HardState { term: 0, vote: 0, commit: 0 }, conf_state: ConfState { voters: [1725973526092207], learners: [], voters_outgoing: [], learners_next: [], auto_leave: false } }, latest_snapshot_meta: SnapshotMetadataSer { term: 0, index: 0 }, apply_progress_queue: EntryApplyProgressQueue(None), peer_address_by_id: RwLock { data: {} }, peer_metadata_by_id: RwLock { data: {} }, cluster_metadata: {}, this_peer_id: 1725973526092207, path: "./storage/raft_state.json", dirty: false }
2024-10-16T02:34:56.831912Z  INFO qdrant: Distributed mode disabled
2024-10-16T02:34:56.832015Z  INFO qdrant: Telemetry reporting enabled, id: d2a1870b-8953-433d-8cf6-1b5cb6abd0fb
2024-10-16T02:34:56.832341Z DEBUG qdrant: Waiting for thread web to finish
2024-10-16T02:34:56.832861Z  WARN qdrant::actix::web_ui: Static content folder for Web UI './static' does not exist
2024-10-16T02:34:56.833253Z  INFO qdrant::actix: TLS disabled for REST API
2024-10-16T02:34:56.834346Z  INFO qdrant::actix: Qdrant HTTP listening on 6333
2024-10-16T02:34:56.834370Z  INFO actix_server::builder: Starting 1 workers
2024-10-16T02:34:56.834394Z  INFO actix_server::server: Actix runtime found; starting in Actix runtime
2024-10-16T02:34:56.841465Z DEBUG reqwest::connect: starting new connection: https://telemetry.qdrant.io/
2024-10-16T02:34:56.845585Z  INFO qdrant::tonic: Qdrant gRPC listening on 6334
2024-10-16T02:34:56.845666Z  INFO qdrant::tonic: TLS disabled for gRPC API

とりあえずQdrantが起動するところまで確認できたので、次はダッシュボードから簡単な操作を試してみます。ダッシュボード用のアセット類はQdrantのリポジトリ本体には含まれていないので、リポジトリに含まれているsync-web-ui.shというシェルスクリプトを使ってセットアップします。

./tools/sync-web-ui.sh

以下のように出力されればOKです。

--2024-10-16 02:35:36--  https://github.com/qdrant/qdrant-web-ui/releases/download/v0.1.33/dist-qdrant.zip
Resolving github.com (github.com)... 140.82.112.4
Connecting to github.com (github.com)|140.82.112.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/607218649/2d6999a9-c95e-402d-81df-13da7a0a19e2?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241016%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241016T023536Z&X-Amz-Expires=300&X-Amz-Signature=6c8fc58409a1dcb04b1a716fba3dd04362279d165a473c20a036418d1ebca9e6&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Ddist-qdrant.zip&response-content-type=application%2Foctet-stream [following]
--2024-10-16 02:35:36--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/607218649/2d6999a9-c95e-402d-81df-13da7a0a19e2?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241016%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241016T023536Z&X-Amz-Expires=300&X-Amz-Signature=6c8fc58409a1dcb04b1a716fba3dd04362279d165a473c20a036418d1ebca9e6&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Ddist-qdrant.zip&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.109.133, 185.199.110.133, 185.199.111.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.109.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2134043 (2.0M) [application/octet-stream]
Saving to: ‘dist-qdrant.zip’

dist-qdrant.zip                                          100%[===============================================================================================================================>]   2.04M  --.-KB/s    in 0.01s

2024-10-16 02:35:36 (159 MB/s) - ‘dist-qdrant.zip’ saved [2134043/2134043]

Archive:  dist-qdrant.zip
   creating: ./static/dist/
  inflating: ./static/dist/robots.txt
  inflating: ./static/dist/logo512.png
   creating: ./static/dist/assets/
  inflating: ./static/dist/assets/scala-9222416a.js
  ...略
  inflating: ./static/dist/qdrant-web-ui.spdx.json
  inflating: ./static/dist/logo.png

続いてインターネット経由でダッシュボードにアクセスできるよう設定ファイルを用意します。

※ 前提としてセキュリティグループ等でアクセス元のIPが適切に設定されていることとします。

設定ファイルは優先順位に従って読み込まれるので、このファイルをconfig/local.yamlに配置すればデフォルトの各種設定を上書きしてくれます。

log_level: DEBUG

service:
  host: 0.0.0.0
  http_port: 6333
  grpc_port: 6334

storage:
  performance:
    max_search_threads: 4

  optimizers:
    flush_interval_sec: 5

    default_segment_number: 2

  handle_collection_load_errors: true

再度Qdrantを起動し、

http://EC2のGIP:6333/dashboardにアクセスし、ダッシュボードが表示されることを確認してみましょう。

Qdrantのダッシュボード

無事にダッシュボードが表示できました。これでQdrant on EC2の環境構築完了です。

参考までにQdrantビルド時のエラー

ちなみに必要なライブラリ等がインストールされていない状態でcargo buildするとビルドエラーになります。トライ&エラーの過程で発生したエラーをいくつか紹介します。

gccがインストールされていない場合

error: linker `cc` not found
  |
  = note: No such file or directory (os error 2)

error: could not compile `proc-macro2` (build script) due to 1 previous error
warning: build failed, waiting for other jobs to finish...

clang未インストールの場合

error: failed to run custom build command for `librocksdb-sys v0.16.0+8.10.0`

Caused by:
  process didn't exit successfully: `/home/ec2-user/qdrant/target/release/build/librocksdb-sys-27289ac554c36a73/build-script-build` (exit status: 101)
  --- stderr
  thread 'main' panicked at /home/ec2-user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen-0.69.4/lib.rs:622:31:
  Unable to find libclang: "couldn't find any valid shared libraries matching: ['libclang.so', 'libclang-*.so', 'libclang.so.*', 'libclang-*.so.*'], set the `LIBCLANG_PATH` environment variable to a path where one of these files can be found (invalid: [])"
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...

protobuf-compiler未インストールの場合

error: failed to run custom build command for `prost-wkt-types v0.5.1`

Caused by:
  process didn't exit successfully: `/home/ec2-user/qdrant/target/release/build/prost-wkt-types-20e90f1289fde0bf/build-script-build` (exit status: 101)
  --- stderr
  thread 'main' panicked at /home/ec2-user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/prost-wkt-types-0.5.1/build.rs:47:10:
  called `Result::unwrap()` on an `Err` value: Custom { kind: NotFound, error: "Could not find `protoc`. If `protoc` is installed, try setting the `PROTOC` environment variable to the path of the `protoc` binary. To install it on Debian, run `apt-get install protobuf-compiler`. It is also available at https://github.com/protocolbuffers/protobuf/releases  For more information: https://docs.rs/prost-build/#sourcing-protoc" }
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...

protobuf-devel未インストールの場合

error: failed to run custom build command for `prost-wkt-types v0.5.1`

Caused by:
  process didn't exit successfully: `/home/ec2-user/qdrant/target/release/build/prost-wkt-types-20e90f1289fde0bf/build-script-build` (exit status: 101)
  --- stderr
  thread 'main' panicked at /home/ec2-user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/prost-wkt-types-0.5.1/build.rs:47:10:
  called `Result::unwrap()` on an `Err` value: Custom { kind: Other, error: "protoc failed: google/protobuf/duration.proto: File not found.\ngoogle/protobuf/timestamp.proto: File not found.\npbtime.proto:3:1: Import \"google/protobuf/duration.proto\" was not found or had errors.\npbtime.proto:4:1: Import \"google/protobuf/timestamp.proto\" was not found or had errors.\n" }
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...

この場合、エラーログは出力されるもののビルド自体は継続していたので、もしかすると無視しても問題ないのかもしれません。私はCtrl + Cでビルドを中断後にprotobuf-develをインストールしてやり直しました。

OOM Killerにkillされた場合

依存関係が問題なくてもメモリ不足でビルドが失敗することがありました。その場合はビルドのプロセスがOOM Killerにkillされて、以下のようなエラーメッセージが出力されます。

error: could not compile `qdrant` (bin "qdrant")

Caused by:
  process didn't exit successfully: `/home/ec2-user/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/rustc --crate-name qdrant --edition=2021 src/main.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=225 --crate-type bin --emit=dep-info,link -C opt-level=3 -C lto=fat -C codegen-units=1 '--warn=clippy::used_underscore_binding' '--warn=clippy::unused_self' '--warn=clippy::unnecessary_wraps' '--warn=clippy::uninlined_format_args' --warn=unexpected_cfgs '--warn=clippy::ref_option_ref' '--warn=clippy::ptr_as_ptr' '--warn=clippy::needless_raw_string_hashes' '--warn=clippy::needless_continue' '--warn=clippy::manual_let_else' '--warn=clippy::manual_is_variant_and' '--warn=clippy::inefficient_to_string' '--warn=clippy::inconsistent_struct_constructor' '--warn=clippy::implicit_clone' '--warn=clippy::from_iter_instead_of_collect' '--warn=clippy::flat_map_option' '--warn=clippy::filter_map_next' '--warn=clippy::explicit_into_iter_loop' '--warn=clippy::enum_glob_use' '--warn=clippy::doc_link_with_quotes' '--warn=clippy::cast_lossless' --cfg 'feature="actix-web"' --cfg 'feature="default"' --cfg 'feature="parking_lot"' --cfg 'feature="web"' --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values("actix-web", "chaos-testing", "console", "console-subscriber", "data-consistency-check", "default", "multiling-chinese", "multiling-japanese", "multiling-korean", "parking_lot", "rstack-self", "service_debug", "stacktrace", "tokio-tracing", "tracing", "tracing-tracy", "tracy", "web"))' --check-cfg 'cfg(tokio_unstable)' -C metadata=c352753418128bbc -C extra-filename=-c352753418128bbc --out-dir /home/ec2-user/qdrant/target/release/deps -C strip=debuginfo -L dependency=/home/ec2-user/qdrant/target/release/deps --extern actix_cors=/home/ec2-user/qdrant/target/release/deps/libactix_cors-7ef02b9d13f4dc03.rlib --extern actix_files=/home/ec2-user/qdrant/target/release/deps/libactix_files-77a36d926af4ab15.rlib --extern actix_multipart=/home/ec2-user/qdrant/target/release/deps/libactix_multipart-3863883f92cdddae.rlib --extern actix_web=/home/ec2-user/qdrant/target/release/deps/libactix_web-1e18370f66b14de4.rlib --extern actix_web_extras=/home/ec2-user/qdrant/target/release/deps/libactix_web_extras-90a03c042437620e.rlib --extern actix_web_validator=/home/ec2-user/qdrant/target/release/deps/libactix_web_validator-c3cfacab85c9c36e.rlib --extern anyhow=/home/ec2-user/qdrant/target/release/deps/libanyhow-c5df93f7e66c6fca.rlib --extern api=/home/ec2-user/qdrant/target/release/deps/libapi-7846a53ba5d9da38.rlib --extern bytes=/home/ec2-user/qdrant/target/release/deps/libbytes-df8cf01997630810.rlib --extern cancel=/home/ec2-user/qdrant/target/release/deps/libcancel-d752100b30dd5246.rlib --extern chrono=/home/ec2-user/qdrant/target/release/deps/libchrono-14764882c6c5b1d8.rlib --extern clap=/home/ec2-user/qdrant/target/release/deps/libclap-fc219e9a9e2708a9.rlib --extern collection=/home/ec2-user/qdrant/target/release/deps/libcollection-f3e083807a24bc95.rlib --extern colored=/home/ec2-user/qdrant/target/release/deps/libcolored-3bd1d8fade25e02a.rlib --extern common=/home/ec2-user/qdrant/target/release/deps/libcommon-aebdebfbe26e9d7a.rlib --extern config=/home/ec2-user/qdrant/target/release/deps/libconfig-98dd8d685bce6ea1.rlib --extern constant_time_eq=/home/ec2-user/qdrant/target/release/deps/libconstant_time_eq-17f1ee5178760ad6.rlib --extern futures=/home/ec2-user/qdrant/target/release/deps/libfutures-fbb3e392e2b66d0d.rlib --extern futures_util=/home/ec2-user/qdrant/target/release/deps/libfutures_util-7c98af4139a4f6dd.rlib --extern issues=/home/ec2-user/qdrant/target/release/deps/libissues-0c725a83f4d66523.rlib --extern itertools=/home/ec2-user/qdrant/target/release/deps/libitertools-49ca07852e1130b0.rlib --extern jsonwebtoken=/home/ec2-user/qdrant/target/release/deps/libjsonwebtoken-9939b3cfbe5d55b9.rlib --extern log=/home/ec2-user/qdrant/target/release/deps/liblog-1632e44412d424b3.rlib --extern memory=/home/ec2-user/qdrant/target/release/deps/libmemory-6cb7faf1dcb68d76.rlib --extern parking_lot=/home/ec2-user/qdrant/target/release/deps/libparking_lot-f0837bc5f4d4814e.rlib --extern prometheus=/home/ec2-user/qdrant/target/release/deps/libprometheus-a41dcf76b797e0d5.rlib --extern prost_for_raft=/home/ec2-user/qdrant/target/release/deps/libprost-f1cce85c48903d38.rlib --extern pyroscope=/home/ec2-user/qdrant/target/release/deps/libpyroscope-3ce6d6a78d1f2d79.rlib --extern pyroscope_pprofrs=/home/ec2-user/qdrant/target/release/deps/libpyroscope_pprofrs-a296edbbbb5390da.rlib --extern raft=/home/ec2-user/qdrant/target/release/deps/libraft-6c8067a46a73675a.rlib --extern raft_proto=/home/ec2-user/qdrant/target/release/deps/libraft_proto-ace6149d5214be20.rlib --extern rand=/home/ec2-user/qdrant/target/release/deps/librand-d06daf4d6da36a10.rlib --extern reqwest=/home/ec2-user/qdrant/target/release/deps/libreqwest-4a38484019003358.rlib --extern rustls=/home/ec2-user/qdrant/target/release/deps/librustls-b33494cae9d00fb6.rlib --extern rustls_pemfile=/home/ec2-user/qdrant/target/release/deps/librustls_pemfile-64b21d9384fd0163.rlib --extern rustls_pki_types=/home/ec2-user/qdrant/target/release/deps/librustls_pki_types-9d5efecd8ed68782.rlib --extern schemars=/home/ec2-user/qdrant/target/release/deps/libschemars-fea60bbc633ff0a6.rlib --extern segment=/home/ec2-user/qdrant/target/release/deps/libsegment-a9466d17958513b6.rlib --extern serde=/home/ec2-user/qdrant/target/release/deps/libserde-948cd0b6012e2ff5.rlib --extern serde_cbor=/home/ec2-user/qdrant/target/release/deps/libserde_cbor-6399556742208cdd.rlib --extern serde_json=/home/ec2-user/qdrant/target/release/deps/libserde_json-d6b4ac7122c518c3.rlib --extern slog=/home/ec2-user/qdrant/target/release/deps/libslog-ee2e0297521ac664.rlib --extern slog_stdlog=/home/ec2-user/qdrant/target/release/deps/libslog_stdlog-3a5cbf8d81631565.rlib --extern storage=/home/ec2-user/qdrant/target/release/deps/libstorage-44e4d1677c387766.rlib --extern sys_info=/home/ec2-user/qdrant/target/release/deps/libsys_info-5f6fcbddbc873f2b.rlib --extern thiserror=/home/ec2-user/qdrant/target/release/deps/libthiserror-81dcc120b63b0b6e.rlib --extern tikv_jemallocator=/home/ec2-user/qdrant/target/release/deps/libtikv_jemallocator-f3ca4cb0a9832af3.rlib --extern tokio=/home/ec2-user/qdrant/target/release/deps/libtokio-a25e4267722fef4a.rlib --extern tokio_util=/home/ec2-user/qdrant/target/release/deps/libtokio_util-b1b615bba787226c.rlib --extern tonic=/home/ec2-user/qdrant/target/release/deps/libtonic-b8f5fb5a72858335.rlib --extern tonic_reflection=/home/ec2-user/qdrant/target/release/deps/libtonic_reflection-7d4713c4f75e67fe.rlib --extern tower=/home/ec2-user/qdrant/target/release/deps/libtower-9b2e44dd36b42f3e.rlib --extern tower_layer=/home/ec2-user/qdrant/target/release/deps/libtower_layer-94036b01e3161e26.rlib --extern tracing=/home/ec2-user/qdrant/target/release/deps/libtracing-16f0cf476e179889.rlib --extern tracing_log=/home/ec2-user/qdrant/target/release/deps/libtracing_log-f8c7b05a7e755f6d.rlib --extern tracing_subscriber=/home/ec2-user/qdrant/target/release/deps/libtracing_subscriber-f2eb94e8420271a0.rlib --extern uuid=/home/ec2-user/qdrant/target/release/deps/libuuid-653682dfcd4732a3.rlib --extern validator=/home/ec2-user/qdrant/target/release/deps/libvalidator-108b506351c62b8a.rlib --extern wal=/home/ec2-user/qdrant/target/release/deps/libwal-8088d52dbb8157bc.rlib -L native=/home/ec2-user/qdrant/target/release/build/ring-351c4b79e851e656/out -L native=/home/ec2-user/qdrant/target/release/build/zstd-sys-bb0f3a8203861ff0/out -L native=/home/ec2-user/qdrant/target/release/build/quantization-6306d3cdc957c576/out -L native=/home/ec2-user/qdrant/target/release/build/librocksdb-sys-ec2ce6ee4c1feef3/out -L native=/home/ec2-user/qdrant/target/release/build/librocksdb-sys-ec2ce6ee4c1feef3/out -L native=/home/ec2-user/qdrant/target/release/build/lz4-sys-32139a8a81d23927/out -L native=/home/ec2-user/qdrant/target/release/build/ring-4d11043b308c7ca4/out -L native=/home/ec2-user/qdrant/target/release/build/sys-info-81d163270638fac7/out -L native=/home/ec2-user/qdrant/target/release/build/tikv-jemalloc-sys-73c639b73fedf667/out/build/lib` (signal: 9, SIGKILL: kill)

sudo journalctl -p err | grep -i "out of memory"でログを確認すると、以下のようにOOM Killerが発動していることが分かります。

Oct 16 01:09:46 ip-172-31-4-155.ec2.internal kernel: Out of memory: Killed process 39954 (rustc) total-vm:10940020kB, anon-rss:7232152kB, file-rss:1664kB, shmem-rss:0kB, UID:1000 pgtables:19176kB oom_score_adj:0